1. Field of the Invention
The present invention relates to a novel method and system for 3 d-aided-2D face recognition.
More particularly, the present invention relates to a novel method and a system for 3D-aided 2D face recognition under large pose and illumination variations, where the method includes enrolling a face of a subject into a gallery database, where the enrollment data comprises either data from a 3D scanner or imaging device or data derived from a 2D imaging or scanning device, raw 3D data. The method also includes verifying and/or identifying a target face form data produced by a 2D imagining or scanning device, 2D data. During the enrollment process, a statistically derived annotated face model is fitted using a subdivision-based deformable model framework to the raw 3D data. The annotated face model is capable of being smoothly deformed into any face so it acts as a universal facial template. During authentication or identification, only a single 2D image is required. The subject specific fitted annotated face model from the gallery is used to lift a texture of a face from a 2D probe image, and a bidirectional relighting algorithm is employed to change the illumination of the gallery texture to match that of the probe. Then, the relit texture is compared to the gallery texture using a view-dependent complex wavelet structural similarity index metric. The inventors have shown that using this approach yields significantly better recognition rates, with an equal error rate (EER) which is less than half (12.1%) of the best performing, commercially available 2D face recognition software. The term texture as used herein means the specific coloring and other facially distinct elements of the face as seen from the 2D imagining device. It is this data that is lifted and pasted onto the subject fitted annotated face model.
2. Description of the Related Art
The human face, as the most distinctive and descriptive human feature, has been widely researched in both the computer vision and computer graphics domains. With the proliferation of 3D scanners, 3D facial data are used in biometrics, motion pictures, games and medical applications.
However, the use of real facial data introduces a number of challenges. The texture acquired by sensors is affected by the lighting conditions. In applications such as biometrics or face relighting the skin albedo is required. Therefore, the contribution of environmental lighting must be removed from the texture. Moreover, commercial 3D scanners produce 3D data that have artifacts and non-uniform sampling. Manual cropping is required to remove data that do not belong to the face. Finally, there is no common point of reference between facial scans even of the same person, as these data are unregistered.
3D-aided 2D face recognition has its roots in the work on estimating the 3D structure of human faces from 2D images using 3D morphable models by Vetter and Blanz[13,14]. The first implementation required a lot of interaction from the user, who had to select a large number of points on the 2D image and the corresponding points on the 3D model. Initially, the algorithms only worked at very specific angles and illumination conditions. Over time, the number of points needed to be manually matched has been reduced significantly[15]. The morphable model employed in estimating the 3D structure of faces is created from 3D scans of multiple humans. Theoretically, if many 3D scans are available, the structure of any face can be represented as a linear combination of the faces in the database. Since the number of faces in the database may be quite large, the authors employ Principal Components Analysis in order to determine the most important features that describe a face or the 3D “eigenfaces”. This approach allows the authors to remove from the database those faces that do not contribute much to the variation between humans (e.g., if two faces are similar, only one most likely be used in the final database). The first reliable 3D-aided 2D face recognition algorithms were an extension of the original work of Vetter and Blanz[16], and it included lighting estimation, because the most frequent reason for which the initial attempts had failed was due to lighting. A more advanced lighting estimation technique was developed using spherical harmonics to further improve the results obtained using the morphable model approach[17,18]. The morphable model approach was also extended to work with non-frontal images[19]. The morphable model 3D-aided 2D face recognition approach still requires carefully selected (manually) feature points and consequently it is still not ready to be used in an operational scenario. Lee and Ranganath developed a different approach to perform 3D/2D face recognition[20]. Instead of using a morphable model created from 3D scans of faces, the authors used a generic 3D model deformed using certain parameters. They used this model in the same fashion as the previous approaches, but they did not estimate lighting. They compensated for this by making use of edge matching. The performance reported is not satisfactory for the requirements of an operational scenario. Classical 2D face recognition algorithms that have been trained using 3D faces rendered under different illumination conditions represent another class of methods that does not require user interaction. This idea has been applied to face recognition using component-based Support Vector Machines[21] and Fisherfaces[22]. The main problem with this class of methods still is lighting. The results reported demonstrate that more research is needed to explore this class of methods.
In summary, neither morphable models nor 2D face recognition algorithms trained using synthetic data perform well enough or are practical enough to be used in a face recognition system deployed in an operational scenario. Clearly, a different approach must be taken to alleviate the challenges of face recognition.
Thus, there is a need in the art for an automated method for human face modeling and relighting with application to face recognition.